contextdb v2: a search index for vector columns - #22

Adds a search index for vector columns to the workspace under v2/. Nothing that builds from crates/ changes; the root Cargo.toml and Cargo.lock are untouched. One commit on dev c79bdee.

What is in it

• An index a column grows into. A vector column is searched exactly until it holds AUTO_INDEX_AT vectors (100,000 unless declared) or declares SEARCH_MODE INDEXED. Until then no centre and no sketch is written, so a small store stays the size it was. From there each partition's sealed vectors are grouped around centres, and every vector gets a one-bit-per-number sketch that a search reads before the vectors themselves.
• How a search uses it. It ranks the centres, takes the groups whose centre is within a measured distance of the best one, estimates their sketches and those of the rows sealed since the build, passes what the sketches keep through the one-byte screen of the previous change, and decides by true distance. Every row of an answer carries its true distance. Three modes: USE VECTOR EXACT, USE VECTOR INDEXED, and AUTO, which is what a statement gets when it names none.
• What an indexed answer promises. How far the scan reaches and how far a sketch is trusted are measured when the index is built, on questions made from the stored vectors: half are stored vectors, half lie between two. A row counts as kept only when its sketch covers the row's own true similarity, both as a grouped row and as a row written after the build, and at least 99 in 100 of those questions' ten nearest rows must be kept. This is a measure, not a promise for any one question. USE VECTOR EXACT is exact.
• Filters. A filter on rows is asked of each row the search visits, and the search widens until it holds as many allowed rows as were asked for. When fewer rows pass than LIMIT, AUTO finishes as an exact search and INDEXED is refused by name.
• Building is a call the host makes. run_maintenance_cycle seals new vectors as before and now builds a partition's index in bounded steps, switching it in with one commit. A kill at any storage call leaves every vector found. Rows written meanwhile are searched through their own sketches until the next build takes them in. A step that does not fit the memory limit is recorded as stalled and the index in force keeps serving.
• Reduced vectors. quantization = 'SQ8' stores a table's vectors at one byte a number; it is an option a table chooses, never a default. SQ4 is refused by name.
• Inspection. SHOW VECTOR_INDEXES, SHOW VECTOR_PARTITIONS, Database::explain and the shell's .explain say what a column holds and what a search would do.
• Damage. A grouping or sketch record that disagrees with itself is reported as damage, never a panic; under AUTO the search answers exactly in its place.
• The check. .github/workflows/check.yml states 15,500 shipped lines and 260 tests (it stated 11,000 and 180).

Measured at this commit

By v2/numbers.sh: a release build on a 16-core AMD Ryzen 9 9955HX with 64 GB of memory and an NVMe drive, with nothing else running. One query at a time on one thread, a read session at 1 GiB, 95th percentile of 100 questions. Recall is the share of the exact answer's rows that the indexed answer returned; "all ten" counts the questions whose ten rows were all the same. The library picks the processor's bit-count instruction when it runs, and these times are with it.

Against the current engine, like for like

The same rows loaded clean into both engines, 500 to a commit, into the same table: text passages with one 768-number embedding each, in 8 partitions (a real workload that is not in the repository). The same 100 questions, a read session at 1 GiB, one query at a time, one core each, 95th percentile, same machine. The current engine and this change's exact search return the same ten rows for every question; this change's indexed search returns them at the recall above.

Verification

• Every command of check.yml as written, on the commit, plus the build on Rust 1.89 and a cross-compiled lint for aarch64-apple-darwin: all pass. 234 of 234 tests and the documentation tests; the storage library's 202; 15,443 shipped lines against the stated 15,500 and 253 tests against the stated 260.
• GitHub Actions on 718698cda: the run is linked in a comment below when it finishes.
• Read by two other model families before this commit, and their findings fixed: a block sealed while a partition's first build starts was left without sketches; damaged stored bytes that could panic or pass unreported in four places; a visited row counted twice, which sent most searches around a second time; a filtered search that could lose an allowed row; an accuracy measure that was too lenient; a build step that returned the memory refusal where it should record a stall; a wrong line in explain; memory taken before it was asked for. The changes made after the second read were not read again by another model family.
• Two of those fixes have no test that separates them (the row counted once, and the threshold that rows written after a build are held against). The passes_mean, sketches_estimated_mean and tail lines of v2/numbers.sh are what shows them.

Known limits of this version

• Nothing seals or builds on its own; a host calls run_maintenance_cycle. Before a build a search returns the exact answer, slower.
• Right after an index is built the file is larger than the index adds: 1.13 times the sealed store at 100,008 passages and 1.31 times at a million vectors. The freed pages are given back by a later change.
• The slowest questions cost far more than the middle one where a question lies near no group: 30 ms against 0.6 ms on a million generated sightings.
• The accuracy measure is taken at build time, on the build's own questions, at ten rows. A question unlike both kinds, rows written after a build that are unlike the rows the build saw, and a LIMIT above ten are not what it measured.
• The reference that sketches of rows written after a build are taken from (a partition's mean) is set at the partition's first build and kept.
• On small or low-dimension stores an indexed search can make more distance evaluations than an exact one; it still reads fewer bytes.
• A condition that passes few rows is searched through the vector index. Reading the condition's own index first comes with the query language.
• EXPLAIN SELECT is not accepted as SQL; .explain and Database::explain are the doors. A VECTOR(n) column's clauses must be written in one order.
• A reduced (SQ8) vector kept inside its row is expanded when the row is read, before the read's memory is asked for.